os: start processes with posix_spawn - #5634
Open
yohimik wants to merge 5 commits into
Open
Conversation
This was referenced Aug 30, 2026
yohimik
force-pushed
the
upstream-pr/os-exec-posix-spawn
branch
from
September 2, 2026 08:50
628c5e6 to
42486b3
Compare
Author
|
Rebased on dev after the 0.42.0 release. The change applies on top of v0.42.0 |
Use posix_spawn on hosted Linux and Darwin. Map process files, apply the working directory and process group, and clear the child signal mask. Use wait4 for process status and support Kill and Signal. Mark Darwin pipes close-on-exec under ForkLock. Darwin also needs the fcntl wrapper in PR tinygo-org#5612 and the libSystem symbols in PR tinygo-org#5636. Concurrent spawn and pipe creation need the RWMutex fix in PR tinygo-org#5630. Keep the process stubs on other targets and add process regression tests.
yohimik
force-pushed
the
upstream-pr/os-exec-posix-spawn
branch
from
September 5, 2026 02:28
42486b3 to
8af7034
Compare
This was referenced Sep 5, 2026
Open /dev/null in the child before closing a nil or unnamed descriptor. Save remap sources before these actions. This avoids a parent descriptor check and its race with concurrent open. Use descriptor reads and exec redirection in remap tests. Check nil slots and temporary descriptors. The tests pass with ordinary Go on Darwin and Linux. Both TinyGo os suites pass with released compiler source overlays. Darwin also needs the addopen symbol from PR tinygo-org#5636.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
os: start processes with posix_spawn
Implement process creation and waiting on hosted Linux and Darwin.
Darwin needs the fcntl wrapper fix in #5612 and the libSystem symbols in #5636.
Concurrent process creation and pipe creation also need #5630.
Changes
The old process layer refused
ProcAttr.Files,Dir, andSys.os/execsuppliesFiles, so it could not start commands.posix_spawnso that Go code does not run in a child afterfork.This avoids inherited runtime and collector locks.
ProcAttr.Filesto child descriptors. Close nil entries and unnamedstandard descriptors. Apply
Dirwithaddchdir_np./dev/nullin each nil or unnamed slot,then close that slot. This makes the close valid even if the descriptor was
already closed. Both actions run in the child after sources have been saved.
There is no parent descriptor check or mutation.
This handles cycles, repeated sources, and sources below their target index.
Temporary descriptors are above all sources and targets. They exist only
in the child and are closed after the mappings. Parent descriptors stay intact.
Envis nil and clear the child signal mask.SysProcAttr.SetpgidandPgid. Refuse other nonzero fields by namewith an error that wraps
ErrNotImplementedSys.wait4, retry EINTR, and report the process ID, exit status, and resourceusage. Support
KillandSignal, includingErrProcessDoneafter a reap.syscall.ForkLock.Linux uses
pipe2withO_CLOEXEC.Targets without a process model keep the previous stubs.
Tests
The existing process tests cover exit status, kill, missing programs, working
directory, process groups, refused attributes, pipe inheritance, and closed
standard descriptors.
src/os/exec_remap_test.goadds cycle, repeated-source, closed-source, identity,and sparse mappings. It checks the child mappings, closure of nil slots and
temporary descriptors, and access to the original parent files. It also checks
an invalid source descriptor. The child uses real descriptor reads and
execredirection, not
/dev/fdmetadata. These checks use/bin/bashfor descriptornumbers above 9 and explicitly skip when that shell is absent.
The Linux source-overlay run used Debian 13 with Bash 5.2.37. All five cases
ran. A separate Alpine 3.24.1 check without Bash reported
SKIPfor the mappingtest, even though the binary's final status was
PASS. That is skippedcoverage, not a successful remapping test. The targeted Alpine harness installed
Bash with
apk add --no-cache bash, then ran the same test binary. All fivecases passed with Bash 5.3.9.
The checked Linux workflow uses Alpine for release builds and Ubuntu for
hosted standard-library tests. The listed APK dependencies do not include Bash,
and their simulated dependency resolution on Alpine 3.24.1 did not add it.
No workflow change was made. A targeted Alpine test harness must install Bash
and check the individual test output for skips.
Current follow-up results on Linux arm64 use the released TinyGo 0.42.0 compiler
and Go 1.27.0. A private TINYGOROOT copy contains this branch's source files and
the released compiler's library files. Compilation uses
-p 1andGOMAXPROCS=2.The identity case passes.
tinygo test -p 1 -v -run TestForkExec ospasses.tinygo test -p 1 ossuite passes.The first Darwin run at
8af70341found two defects. The/dev/fdtest checksalso failed with ordinary Go. Separately, Darwin's native
posix_spawnreturnedEBADFfor a close action on an unopened descriptor. With corrected tests butthe old actions, cycle, repeated-source, closed-source, and identity still
failed at StartProcess. Sparse passed.
Commit
84032b325c9343065746e367119b92ff6bab0600corrects both issues.The corrected remapping tests first passed with ordinary Go 1.26.7 on Darwin
arm64 and Go 1.27.0 on Linux arm64. With the new file actions, both the focused
process tests and the full
ospackage pass on both systems.Darwin validation uses macOS 26.6.2, the released
0.43.0-net.1compiler,and Go 1.26.7. A private TINYGOROOT copy contains the new
src/os. Its privateSDK assembly also has the added
_posix_spawn_file_actions_addopensymbolstub. The shared release and cache are unchanged. The compiler and runtime
remain from the release and contain the older fcntl correction.
Commands were
tinygo test -p 1 -v -run TestForkExec osandtinygo test -p 1 os, withGOMAXPROCS=2and external 180-second limits.Darwin completed the focused tests in 0.590 s and
osin 0.795 s. Linuxcompleted them in 0.017 s and 0.060 s. No remapping cases were skipped.
These are released-compiler source-overlay results. They do not validate a
compiler rebuilt from #5612 and #5636, the combined fork candidate, or a new
downstream full suite. Fork PR #18
owns combined CI. Canceled validation-only
PR #16 runs are not successful validation.
Downstream context
Dispat uses the related hosted process, signal, network, and TLS work. Its
TinyGo acceptance record
describes the downstream integration history. That record is separate from
the focused results for this PR and is not a new downstream full-suite result.
Published fork and downstream evidence
The net.2 fork release combines the coordinated changes at
95fba82a, with net0f460803. It differs from accepted candidatee7d34c8conly in the version constant. Linux, macOS and Windows branch CI and tag CI passed on their first attempts.Dispat source
909dc401, with harness0990c6db, passed 796 test events with no failures or skips on each native Darwin ARM64 and Linux ARM64 candidate run. Crier source7d687fc8passed raw and stripped E2E on native Linux ARM64 and emulated Linux AMD64, each with 144 top-level tests and 156 passing events, no failures or skips. Both applications use TinyGo-built update fixtures and test trusted TLS, certificate refusal, original backup hashes and byte-identical offline rollback. Their workflows also exercise files, environment variables, concurrency and child processes. Crier includes real FFmpeg and webrender/canvas rendering.Crier's unchanged 13-image pixel gate passed. It uses an approved two-line explicit-rounding webrender build patch for both compilers. The earlier gradient mismatch was permitted fused arithmetic, not a TinyGo compiler error. Candidate stripped sizes are 13,835,824 versus 30,277,794 Go bytes on ARM64 (54.30% smaller), and 16,446,968 versus 32,518,306 on AMD64 (49.42% smaller).
These are combined-candidate application results, not proof that this PR alone supplies the features. They supersede the earlier Crier comparison. Published-toolchain probes and application acceptance have since completed. The final Crier v1.1.1 release evidence is below. Dispat controls its own publication. WaitDelay, in-flight deadlines and full descriptor lifetime remain open. This enables tested CLI client workflows, not general Go or server compatibility.
WaitDelay remains open
The manual test in
testdata/os-exec-waitdelayreproduces the reported failurewith the real Go
os/execpackage. A shell exits while a backgroundsleepkeeps the output pipe open for two seconds.
With a 100 ms WaitDelay, TinyGo returns
exec.ErrWaitDelayafter 2.004 seconds.Standard Go returns that error after 101 ms. In a separate pipe test, closing
the reader leaves TinyGo's active read blocked until the writer closes about
two seconds later. Standard Go stops the read immediately.
TinyGo's hosted file layer uses blocking
syscall.Readandsyscall.Close.After the timer expires,
Cmd.awaitGoroutinescloses the pipes and waits forthe copy goroutines. The blocked read prevents that wait from finishing.
The fd mapping fix does not resolve this failure.
A follow-up needs interruptible pipe I/O, coordination between close and active
I/O, and protection against descriptor reuse. It must test normal exit, context
cancellation, blocked reads and writes, and close on Linux and Darwin.
The manual test and its README include commands and measured results.
Dependencies and overlap
on
upstream/devand contains no fcntl implementation or fcntl tests.The existing tests in commit
9161a8c6e75cb1444c21bee6e720a6605e39bc44,src/os/fcntl_test.go, were offered to compiler: implement go:cgo_import_dynamic and lower Darwin cgo import trampolines (#5604) #5612. compiler,runtime: call fcntl through a C wrapper on darwin #5632 is closed in favor of compiler: implement go:cgo_import_dynamic and lower Darwin cgo import trampolines (#5604) #5612.syscall.ForkLockusessync.RWMutex. This is separate from the WaitDelay failure.and sockets, and the shared builder helper. This PR has no builder changes.
The new close actions specifically need
posix_spawn_file_actions_addopen.builder: declare Darwin socket and process symbols #5636 does not depend on this PR or on a net submodule update.
Other limits
SysProcAttrfields other thanSetpgidandPgidremain unsupported.posix_spawn_file_actions_addchdir_npneeds macOS 10.15 or later, while theconfigured deployment target is lower.
Processhas a four-bytedonefield on targets that do not use processes.Related pull requests
in favor of runtime: deliver signals under the threads scheduler when blocked on I/O #5530.
and its regression tests were offered there.
The crypto/tls: use Go's software TLS on the host target #5645 owner closed that PR in favor of runtime: implement weak.runtime_makeStrongFromWeak #5633 and loader: use the real crypto/tls on hosted linux and darwin #5635.
builder helper, with a direct syscall socket test. It needs no net submodule
update and contains no process, TLS, or weak reference implementation.
Related process work is #4471 and #4377. These related changes and their tests
do not establish that WaitDelay works.
Published Crier v1.1.1 evidence
Crier v1.1.1 is public at source
acac2f0eand uses published TinyGo0.43.0-net.2. Its final acceptance report and SHA-256 manifest identify the exact release bytes. The public tag, asset sizes and report digest were checked. These final sizes supersede the candidate sizes above.The report records 144 top-level tests and 156 passing events for each raw and stripped run on native ARM64 and emulated AMD64, with no failures or skips. It covers real CLI files, environment and concurrent work, child processes, TLS, update/rollback fixtures, uploads, real FFmpeg, and webrender/canvas rendering. The unchanged 13-image gate passes on both targets. AMD64 is exact; ARM64 has 12 exact images and four event-card pixels with channel difference 1. Both compilers use the same explicit-rounding webrender build patch. Standard Go tests, 90.6% coverage, lint and docs also pass.
This is combined-fork application evidence, not isolated proof for this PR or general server support. The generic emulated AMD64
osclosure assertions still fail and also fail with ordinary Go under that emulation; they are not counted as passing. Native AMD64 CI and the final published AMD64netpackage pass. The report retains other platform and deadline/descriptor limits. Tiny binaries are opt-in; normal install/self-update selects standard Go assets.Published Dispat v1.8.1 CLI evidence
Dispat v1.8.1 CLI is public. Its size and SHA-256 manifest records build source
40c58236, Go 1.26.8 and TinyGo0.43.0-net.2. The public asset metadata and manifest digest were checked.These are final published CLI asset sizes, not the earlier candidate measurements. They do not replace the separately identified test evidence or remove known runtime limits. The full release workflow has now completed successfully at the recorded build source, including its Windows, macOS and Ubuntu checks. This does not change the test and platform limits stated above.